Systems and methods for non-contacting interaction with user terminals

ABSTRACT

Systems and methods are provided to enable users to interact with user terminals having a touch screen interface without requiring the user to physically contact a surface of the touch screen interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. patent application Ser. No.63/022,774, filed May 11, 2020, entitled SYSTEMS AND METHODS FORNON-CONTACTING INTERACTION WITH TOUCH SCREEN INTERFACES, the disclosureof which is incorporated herein by reference in its entirety.

BACKGROUND

Touch screen interfaces are becoming ubiquitous as they are increasinglyincorporated into a wide variety of devices. Example devices thatinclude touch screen interfaces are ATMs, kiosks, elevator touch panels,point of sale terminals, airport check-in kiosks, supermarketself-check-out terminals, vending machines, gas pumps, casino games,in-restaurant ordering devices, ticket dispensers, employee kiosks,health assessment kiosks, wayfinding and general information kiosks, andpaid parking machines, among numerous others. A vast majority of suchdevices are in public places and are routinely touched by large volumesof people. Users are becoming increasingly concerned with the possibleexchange of bacteria, germs, viruses, and other contagions throughphysical interaction with commonly touched surfaces, including touchscreen interface surfaces. Touch screen interfaces are rarely, if ever,disinfected, sanitized, or even cleaned. While other user inputtechnologies, such as voice control, may allow at least some degree ofcontactless interaction with certain devices, these technologies areoften not practical to implement nor do they provide an acceptable levelof usability.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be more readily understood from a detaileddescription of some example embodiments taken in conjunction with thefollowing figures:

FIG. 1 depicts an example gesture processing unit in accordance with onenon-limiting embodiment.

FIG. 2 depicts another example gesture processing unit in accordancewith one non-limiting embodiment.

FIG. 3 depicts yet another example gesture processing unit in accordancewith one non-limiting embodiment.

FIG. 4 depicts an example mobile commands processing unit in accordancewith one non-limiting embodiment that enables contactless interactionwith a user terminal via a mobile communications device.

FIG. 5 depicts another mobile commands processing unit in accordancewith one non-limiting embodiment that enables contactless interactionwith a user terminal via a mobile communications device

FIG. 6 depicts another example embodiment of contactless control of auser terminal in accordance with one non-limiting embodiment.

FIG. 7 depicts an example Internet-based mobile commands processing unitin accordance with one non-limiting embodiment.

FIG. 8 depicts another example Internet-based mobile commands processingunit in accordance with one non-limiting embodiment.

FIG. 9 depicts a user terminal that includes a built-in gestureprocessing unit in accordance with one non-limiting embodiment.

FIG. 10 depicts a user terminal that includes a built-in HID/Bluetoothmodule allowing for user input from a coupled mobile communicationsdevice in accordance with one non-limiting embodiment.

FIG. 11 depicts an example user terminal that includes a built-inInternet-based mobile commands processing unit in accordance with onenon-limiting embodiment.

DETAILED DESCRIPTION

Various non-limiting embodiments of the present disclosure will now bedescribed to provide an overall understanding of the principles of thestructure, function, and use of the systems and methods as disclosedherein. One or more examples of these non-limiting embodiments areillustrated in the accompanying drawings. Those of ordinary skill in theart will understand that systems and methods specifically describedherein and illustrated in the accompanying drawings are non-limitingembodiments. The features illustrated or described in connection withone non-limiting embodiment may be combined with the features of othernon-limiting embodiments. Such modifications and variations are intendedto be included within the scope of the present disclosure.

Reference throughout the specification to “various embodiments,” “someembodiments,” “one embodiment,” “some example embodiments,” “one exampleembodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with any embodimentis included in at least one embodiment. Thus, appearances of the phrases“in various embodiments,” “in some embodiments,” “in one embodiment,”“some example embodiments,” “one example embodiment,” or “in anembodiment” in places throughout the specification are not necessarilyall referring to the same embodiment. Furthermore, the particularfeatures, structures or characteristics may be combined in any suitablemanner in one or more embodiments.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware. The term softwareis used expansively to include not only executable code, but also datastructures, data stores, and computing instructions in any electronicformat, firmware, and embedded software. The terms information and dataare used expansively and can include a wide variety of electronicinformation, including but not limited to machine-executable ormachine-interpretable instructions; content such as text, video data,and audio data, among others; and various codes or flags. The termsinformation, data, and content are sometimes used interchangeably whenpermitted by context.

The examples discussed herein are examples only and are provided toassist in the explanation of the systems and methods described herein.None of the features or components shown in the drawings or discussedbelow should be taken as mandatory for any specific implementation ofany of these systems and methods unless specifically designated asmandatory. For ease of reading and clarity, certain components, modules,or methods may be described solely in connection with a specific figure.Any failure to specifically describe a combination or sub-combination ofcomponents should not be understood as an indication that anycombination or sub-combination is not possible. Also, for any methodsdescribed, regardless of whether the method is described in conjunctionwith a flow diagram, it should be understood that unless otherwisespecified or required by context, any explicit or implicit ordering ofsteps performed in the execution of a method does not imply that thosesteps must be performed in the order presented but instead may beperformed in a different order or in parallel.

In accordance with various embodiments, the presently disclosed systemsand methods generally enable users to interact with user terminalshaving a touch screen interface without requiring the user to physicallycontact a surface of the touch screen interface. As used herein, “userterminal” broadly refers to any system, device, or component thatincorporates a touch screen interface as a means for user input.Examples of user terminals include, but are not limited to, ATMs,kiosks, casino games, elevator touch panels, point of sale (POS)terminals, gas pumps, airport check-in kiosks, supermarketself-check-out terminals, vending machines, in-restaurant orderingdevices, ticket dispensers, employee kiosks, health assessment kiosks,and wayfinding and general information kiosks. Accordingly, userterminals in accordance with the present disclosure can have a widevariety of form factors and utilize a variety of different types oftouch screen interfaces and/or other forms of user input interfacesrequiring physical contact by a user.

As described in more detail below, the presently disclosed systems andmethods allow a user to interact with a user terminal throughnon-contacting interactions. In some embodiments, for example, a usercan have non-contacting interactions with a touch screen interface of auser terminal through a series of hand gestures. In other embodiments, auser can have non-contacting interactions with a touch screen interfaceof a user terminal through contacting interactions with the user'smobile communication device. The user's interactions with their mobilecommunications device can be relayed to the user terminal throughnetwork communications. Furthermore, in some embodiments, suchcontactless interaction functionality can be provided as a retrofit kitor add-on system that is incorporated into an existing user terminal,although this disclosure is not so limited.

Referring now to FIG. 1, an example gesture processing unit 100 isdepicted in accordance with one non-limiting embodiment. The gestureprocessing unit 100 can be incorporated into an existing user terminal190 in order to provide non-contact inputs from a user. In someembodiments, for example, the gesture processing unit 100 is positionedinternal to a housing, case, or cabinet of the user terminal 190. Theuser terminal can have an existing power supply 112, a primary display114, and a primary computer 116. The primary display 114 can incorporatea touch screen interface that is used to track a user's physicalinteraction with the surface of the interface. Additionally oralternatively, the user terminal 190 may have other forms of inputdevices, such as a numeric keypad, a mouse, a stylus, a keyboard, pushbuttons, and so forth. The gesture processing unit 100 can allow a userto provide inputs to the user terminal 190 through non-contacting handgestures and without requiring physical contact with any surfaces of theuser terminal 190. Such hand gestures can be executed within the fieldof view of a camera 102 that is mounted to the user terminal 190. Insome embodiments, the gesture processing unit 100 can utilize a videofeed from an existing camera 102 that was installed in the user terminal190 at the time of manufacture. In other embodiments, the user terminal190 may not include a camera 102, or the in-situ camera is not suitablefor the needs of the gesture processing unit 100. In such embodiments acamera 102 can be mounted to the user terminal 190 at the time thegesture processing unit 100 is installed. The camera 102 can be a depthsensor camera as depicted, or any other any suitable type of camera.Furthermore, in some embodiments, the user terminal 190 may havemultiple cameras 102.

The gesture processing unit 100 can be connected to the peripheralmodules of the user terminal 190. In accordance with variousembodiments, the gesture processing unit 100 can have inputs forHDMI/VGA, Bluetooth/WiFi communications, and USB devices, and outputsfor HDMI/VGA and USB, for example. When installed in the user terminal190, the gesture processing unit 100 can be treated as a peripheraldevice by the other computers of the user terminal 190.

The gesture processing unit 100 can be connected to the primary computer116 of the user terminal 190 via USB port to provide users with atouchless approach for interacting with the user terminal 190. As such,a user can interact with the user terminal 190 without being required tophysically touch an input device of the user terminal (such as akeyboard, a mouse, a touchscreen, a keypad, and so forth). Additionally,a video stream can be routed from the primary computer 116 to thegesture processing unit 100 by connecting HDMI/VGA output from theprimary computer 116 to an input port in the gesture processing unit100. The primary display 114 of the user terminal 190 can be connectedto the output HDMI/VGA port on the gesture processing unit 100.

In an idle state, the original video feed from the primary computer 116can be proxied via the gesture processing unit 100, unaltered, and thendisplayed on the primary display 114. An application executed by a CPU106 of the gesture processing unit 100 can actively monitor motion viathe connected depth camera 102. In some embodiments, a gesturerecognizer module 104 using a tensor processing unit (TPU) is utilized,although this disclosure is not so limited. Upon the identification ofan “initiate” gesture by a user of the user terminal 190, for example,the CPU 106 can begin to overlay the original video with asemi-transparent image of a cursor by a video overlay module 110 to aidthe user in navigation.

During a user's contactless interaction with the user terminal 190,relevant gestures can be identified and translated to the equivalenthuman interface device (HID) commands by an HID module 108 of thegesture processing unit 100. The commands can be delivered to theprimary computer 116 via USB connection (or other suitablecommunications), while the video feed overlay module 110 can provide auser of the user terminal 190 with visual feedback for each relevantaction on the primary display 114. By way of non-limiting examples,gestures of the user can cause the cursor to point to variousinteractive elements on the screen that would typically be physicallytouched by a user. When the cursor is pointing to a desired interactiveelement, the user can execute a gesture for selecting or activating theinteractive element.

Referring now to FIG. 2, another example of a gesture processing unit200 is depicted in accordance with one non-limiting embodiment. Similarto FIG. 1, the gesture processing unit 200 can be incorporated into anexisting user terminal 290 in order to provide non-contact inputs from auser. The user terminal can have an existing power supply 212, a primarydisplay 214, and a primary computer 216. The primary display 214 canincorporate a touch screen interface that is used to track a user'sphysical interaction with the surface of the interface. Additionally oralternatively, the user terminal 290 may have other forms of inputdevices, such as a numeric keypad, a mouse, a stylus, a keyboard, pushbuttons, and so forth. The gesture processing unit 200 can allow a userto provide inputs to the user terminal 290 through non-contacting handgestures and without requiring physical contact with any surfaces of theuser terminal 290. Such hand gestures can be executed within the fieldof view of camera 202 that is mounted to the user terminal 290. Thecamera 202 can be a depth sensor camera as depicted, or any othersuitable type of camera. Furthermore, in some embodiments, multiplecameras 202 can be utilized.

The gesture processing unit 200 can be connected to the peripheralmodules of the user terminal 290. In accordance with variousembodiments, the gesture processing unit 200 can have inputs forHDMI/VGA, Bluetooth/WiFi communications, and USB devices, and outputsfor HDMI/VGA and USB, for example. When installed in the user terminal290, the gesture processing unit 200 can be treated as a peripheraldevice by the other computers of the user terminal 290.

Similar to the gesture processing unit 100 of FIG. 1, the gestureprocessing unit 200 can be connected to the primary computer 216 of theuser terminal 290 via USB port to provide users with a touchlessapproach for interacting with the user terminal 290. Additionally, avideo stream can be routed from the primary computer 216 to the gestureprocessing unit 200 by feeding the video output from the primarycomputer 216 to an input port in the gesture processing unit 200. Insome embodiments, a VGA to HDMI converter 211 can be used to convert thesource video. As shown, a video input module 210 can be used to compressthe video and provide the compressed video to the CPU 206 via a cameraserial interface (CSI) bus, for example. The primary display 214 of theuser terminal 290 can be connected to the output HDMI/VGA port on thegesture processing unit 100. As shown, an HDMI to VGA converter 209 canbe used to output video prior to feeding it to the primary display 214.

In an idle state, the original video feed from the primary computer 216can be proxied via the gesture processing unit 200, unaltered, and thendisplayed on the primary display 214. An application executed by the CPU206 of the gesture processing unit 200 can actively monitor motion viathe connected camera 202. In some embodiments, a gesture recognizermodule 204 using a tensor processing unit (TPU) is utilized, althoughthis disclosure is not so limited. Upon identifying an “initiate”gesture by a user of the user terminal 290, for example, the CPU 206 canbegin to overlay the original video with a semi-transparent image of acursor by a video overlay module 210 to aid the user in navigation.

During a user's contactless interaction with the user terminal 290,relevant gestures can be identified and translated to the equivalenthuman interface device (HID) commands by an HID module 208 of thegesture processing unit 200. The commands can be delivered to theprimary computer 216 via USB connection (or other suitablecommunications). The CPU 206 can also modify the video feed such thatvisual feedback for each relevant action on the primary display 214. Byway of non-limiting examples, gestures of the user can cause the cursorto point to various interactive elements on the screen that wouldtypically be physically touched by a user. When the cursor is pointingto a desired interactive element, the user can execute a gesture forselecting or activating the interactive element.

FIG. 3 depicts another example of a gesture processing unit 300 inaccordance with one non-limiting embodiment. Similar to FIG. 2, thegesture processing unit 300 can be incorporated into an existing userterminal 390 in order to provide non-contact inputs from a user. Theuser terminal can have an existing power supply 312, a primary display314, and a primary computer 316. The user terminal 390 can also have acamera 302. The gesture processing unit 300 can be similar to thegesture processing unit 200 (FIG. 2), having a CPU 306, and an HIDmodule 308. In some embodiments, the gesture processing unit 300 canalso include a video input module 310, a VGA to HDMI converter 311, anda HDMI to VGA converter 309, although this disclosure is not so limited.This example embodiment of the gesture processing unit 300 does notnecessarily utilize a tensor processing unit (TPU) for gesturerecognition. In other embodiments, however, the CPU 306 can include aTPU without departing from the scope of the present disclosure. As shownin FIG. 3, the camera signal is fed to the CPU 306 processing. In someembodiments, the camera module 302 can include a depth sensor such thatthe camera signal provided to the CPU 306 includes depth information.Providing depth information in the camera signal can increase theability of the CPU 306 to provide real-time gesture processing withoutnecessarily needing a TPU. Thus, the functionality provided by thegesture processing unit 300 can be similar to the functionality providedby the gesture processing unit 200, but without the need for dedicatedTPU.

Referring now to FIG. 4, a mobile commands processing unit 400 isdepicted in accordance with one non-limiting embodiment that enablescontactless interaction with a user terminal 490 via a mobilecommunications device 450. Similar to the gesture processing units 100,200 and 300, the mobile commands processing unit 400 can be integratedinto an existing user terminal 490. Generally, the mobile commandsprocessing unit 400 can serve to mirror the images presented on aprimary display 414 of the user terminal 490 onto a display screen ofmobile communications device 450 via wireless communications, asschematically depicted in FIG. 4. The user can interact with theirmobile communications device 450 (i.e., via touching the display screen)and such interactions are relayed to the user terminal 490.

The user terminal 490 can have a power supply 412, a primary display414, and a primary computer 416, similar to the user terminal 190, 290,390, described above, for example. Similar to the gesture processingunits 100, 200, 300, for example, the mobile commands processing unit400 can have a CPU 406 and a video overlay module 410. In accordancewith various embodiments, the mobile commands processing unit 400 canhave inputs for HDMI/VGA, Bluetooth/WiFi communications, and USBdevices, and outputs for HDMINGA and USB, for example. When installed inthe user terminal 490, the mobile commands processing unit 400 can betreated as a peripheral device by the other computers of the userterminal 490. Furthermore, while the embodiment depicted in FIG. 4, aswell as other embodiments, depict the use of Bluetooth for the purposesof illustration, it is to be appreciated that any suitable short-rangewireless communication protocol and/or technologies can be used withoutdeparting from the scope of the present disclosure.

The mobile commands processing unit 400 can be connected to a primarycomputer 416 of the user terminal 490 via USB port to provide users witha touchless approach for interacting with the user terminal 490.Additionally, a video stream can be routed from the primary computer 416to the mobile commands processing unit 400 by connecting HDM/VGA outputfrom the primary computer 416 to an input port of the mobile commandsprocessing unit 400. The primary display 414 of the user terminal 490can be connected to the output HDMI/VGA port on the mobile commandsprocessing unit 400. In this embodiment, a HID/Bluetooth module 408 isutilized to communicate with a mobile communications device 450. As isto be appreciated, the mobile communications device 450 can be a smartphone, a wearable (i.e., smart watch), tablet computer, laptop computer,or any other suitable device with a screen and that can receive userinput.

In the idle state, the original video feed from the primary computer 416can be proxied via the mobile commands processing unit 400, unaltered,and then displayed on the primary display 414. An application running onthe CPU 406 of the mobile commands processing unit 400 can activelymonitor user connections. After establishing a secure connection withthe mobile communications device 450, the display that wouldconventionally be displayed on the primary display 414 can be displayedon the mobile communications device 450 instead. Once successfullymirrored, the video display of the primary display 414 can operationallydisplay, for example, the identical content that is displayed on themobile communication device 450. Alternatively, in some embodiments,other content can be provided for display on the primary display 414,such as an advertisement/marketing materials, or a variety of othersuitable communications or messages. In other embodiments, the primarydisplay 414 may be blank.

Once mirroring is complete, users can use their mobile communicationsdevice 450 as a touchpad, and the HID/Bluetooth Module 408 of the mobilecommands processing unit 400 can translate such interaction to theequivalent HID (human interface device) commands. The HID/BluetoothModule 408 can deliver the commands to the primary computer 416 via USBconnection. In some embodiments, the video feed overlay module 410 canprovide a user with a visual feedback of each relevant action.

The mobile communication device 450 can be paired to the user terminal490 via any suitable technique. In one example embodiment, the CPU 406can cause a QR code to be displayed on the primary display 414. The userof the user terminal 490 can capture the QR code with a camera of themobile communications device 450. Capturing the QR code can initiate asecure Bluetooth connection between the mobile communications device 450and the HID/Bluetooth module 408 of the mobile commands processing unit400. Furthermore, the images presented on the display screen of themobile communications device 450 can be replicas of the images thatwould be presented on the primary display 414. In some embodiments,however, the images presented on the display screen of the mobilecommunications device 450 can be modified, customized, or otherwiseadjusted for proper display on the mobile communications device 450.

FIG. 5 depicts another example mobile commands processing unit 500 inaccordance with one non-limiting embodiment. Similar to FIG. 4, themobile commands processing unit 500 can be incorporated into an existinguser terminal 590 in order to provide non-contact inputs from a user.The user terminal can have an existing power supply 512, a primarydisplay 514, and a primary computer 516. The mobile commands processingunit 500 can be similar to the mobile commands processing unit 400 (FIG.4), having a CPU 506, and an HID module 508. In some embodiments, themobile commands processing unit 500 can also include a video inputmodule 510, a VGA to HDMI converter 511, and a HDMI to VGA converter509, although this disclosure is not so limited. This embodiment of thegesture processing unit 300 does not utilize, however, a tensorprocessing unit (TPU) for gesture recognition.

The functionality provided by the mobile commands processing unit 500can be similar to the functionality described above with regard tomobile commands processing unit 400. In one example embodiment, forexample, a user can approach the user terminal 590 and open a specificmobile application on their mobile communications device 550. The mobileapplication can finds the user terminal 590 via Bluetooth connection,for example, and attempt to connect. If the connection was successful,in accordance with some embodiments, the application executing on themobile communications device 550 can provide the user with a choice ofmirror mode or touchpad mode.

If the mirror mode is selected, the mobile application can display, forexample, a viewfinder box on the screen of the mobile communicationsdevice 550. The size of the viewfinder box can have the same aspectratio as the primary display 514 user terminal 590, for example. Theuser can point a rear-facing camera of mobile communications device 550towards the primary display 514, such that the camera feed is displayedin the viewfinder box. The user can then interact with the display ofthe mobile communications device 550 (by touching a location in theviewfinder box that is displaying an interactive element from theprimary display, for example). These interactions with the touchscreenof the mobile communications device 550 can be sent to the mobilecommands processing unit 500 and can be translated to coordinates on theprimary display 514. Based on the user's interaction with thetouchscreen of their mobile communications device 550 the HID module 508can send corresponding commands to the primary computer 516.

If the touchpad mode is selected, the mobile application can present avirtual track pad panel on the touchscreen on the mobile communicationsdevice 550, similar in functionality to a touchpad on a laptop computer,for example. The mobile application of the mobile communications device550 can process user interaction (taps, moves and swipes) with thevirtual track pad panel and send them via the Bluetooth connection tothe HID module 508, for example. The HID module 508 can then send theactions to the primary computer 516 user terminal 590 of the viastandard HID protocol. The HID module 508 can also sends the actions andto the CPU 506 via I2C connection, for example.

Referring now to FIG. 6, an example embodiment of contactless control ofa user terminal 690 is depicted. In this embodiment, the user caninteract with their mobile communication device 650 to control a mouseor pointer on a screen of the user terminal 690, similar to the“touchpad mode” described above, for example. More specifically, thetouchscreen of their mobile communication device 650 can serve as trackpad such that as a user moves their finger along the touchscreen oftheir mobile communication device 650, they will see the pointer on thedevice screen move in parallel. When the user taps or double-taps thetouchscreen of the their mobile communication device 650, it wouldregister on the device screen as a “touch” or “click”, or other suitableaction, on the user terminal 690.

Similar to FIG. 4, a microcontroller module 606 can include a USBHID/Bluetooth and/or WiFi Module. The user can connect their mobilecommunication device 650 to the microcontroller module 606 via suitablecommunication protocol and connection process. An application running onthe microcontroller module 606 can actively monitor users. Users can usethe connected mobile communication device 650 as a touchpad, and themicrocontroller module 606 can translate such interaction to theequivalent HID (human interface device) commands and deliver the HIDcommands to the primary computer of the user terminal 690 via USBconnection. Thus, a cursor provided by the operating system of the userterminal 690 can be controlled by the user through the interaction withthe connected mobile communication device 650 without requiring the userto physically contact the user terminal 690.

Referring now to FIG. 7, an example Internet-based mobile commandsprocessing unit 700 is depicted in accordance with one non-limitingembodiment. Generally, the mobile commands processing unit 700 can be aretrofit kit that can establish a bi-directional connection between amobile communications device 750 and a user terminal 790 via a websocketprotocol. Thus, instead of using a direct Bluetooth connection or a WiFiconnection between the mobile communications device 750 and the userterminal 790, a dedicated, secure bi-directional connection between themobile communications device 750 and the user terminal 790 can beestablished. Using this technique for contactless interaction, the usercould either see a broadcast version of the screen of the user terminal790 (similar to FIG. 4, for example) or the user could instead simplyuse their mobile communications device 750 as a touchpad to interactwith the screen of user terminal 790 (similar to FIG. 6, for example).

The user terminal 790 can be similar to user terminals 190, 490, and 690having a power supply 712, a primary display 714, and a primary computer716. The mobile commands processing unit 700 can be similar to themobile commands processing unit 200 of FIG. 4, having a CPU 706, videooverlay module 710, and an HID module 708.

When a user approaches the user terminal 790 in its idle state, anapplication running on the CPU 706 can display an overlay messagedirecting the user to navigate to a specific website address on theirmobile communications device 750. The primary display 714 can also showa unique QR code (or other unique identifier) displayed next to themessage. The user can then navigate to the website address on theirmobile communications device 750 and be prompted to scan the currentlydisplayed QR code from the primary display 714, or otherwise enter theunique identifier. For example, in some embodiments, a uniquealphanumeric code can be displayed on the primary display 714 of theuser terminal 790 that can be typed into the webpage displayed on themobile communications device 750. Once the unique code is entered, adedicated websocket connection can be established between the mobilecommunications device 750 and the user terminal 790 through a webserver760. The primary display 714 of the user terminal 790 can be set todisplay alternative content, display a black screen, or display amessage that private connection in session, for example.

In accordance with various embodiments, the mobile commands processingunit 700 can utilize a video compression algorithm and only send staticimages of the screen to the mobile communications device 750 to minimizethe latency and reduce the required bandwidth. The website presented atthe mobile communications device 750 can capture the user's interactionsand send them to the web server 760, which in turn sends them to themobile commands processing unit 700. The mobile commands processing unit700 can translate such interactions to the HID command, which isdelivered via USB port to the user terminal 790.

Although FIG. 7, and other embodiments described herein (such as FIG.11, below) depict the use of the websocket protocol for illustrationpurposes, this disclosure is not so limited. Instead, any suitableweb-based communication protocol, or combination of multiple web-basedcommunication protocols can be utilized without departing from the scopeof the present disclosure. By way of non-limiting examples, someembodiments of the present disclosure can utilize a RTP Control Protocol(RTCP), an HTTP Live Streaming (HLS) protocol, among others. Further,some embodiments can utilize, for example, an HLS protocol for videotransmission and a websocket protocol for gesture input transmission.

FIG. 8 depicts another example mobile commands processing unit 800 inaccordance with one non-limiting embodiment. Similar to FIG. 7, themobile commands processing unit 800 can be incorporated into an existinguser terminal 890 in order to provide non-contact inputs from a user.The user terminal can have an existing power supply 812, a primarydisplay 814, and a primary computer 816. The mobile commands processingunit 800 can be similar to the mobile commands processing unit 700 (FIG.7), having a CPU 806, and an HID module 808. In some embodiments, themobile commands processing unit 800 can also include a video inputmodule 810, a VGA to HDMI converter 811, and a HDMI to VGA converter809, although this disclosure is not so limited.

In accordance with one example implementation, a user can scans a QRCode with a mobile communications device 850. The QR code can beassociated with the user terminal 890 (i.e. via a unique identifier ofthe user terminal) and direct a browser of the mobile communicationsdevice 850 to a website when scanned. The website can send the commandto the CPU 806 to generate and show the start session QR code. Thiscommand can be sent via a message broker service 858, such as AWS SQS,for example. The CPU 806 can cause a “start-session QR code” to bepresented on the primary display 814 via video overlay. The mobilecommunications device 850 can scan the “start-session QR code” presentedon the primary display 814 using a QR code scanner. In some embodiments,the QR Code scanner is provided by the website interface.

The website can then send the command to the CPU 806 to start thesession. This command is sent via the message broker service 858, forexample. Once the session is started, the CPU 806 can remove thestart-session QR code from the primary display 814 and start sendingframes with original images of the primary display 814 to the websitevia a websocket connection. In some embodiments, the frames are jpegimages, which form an mjpeg stream. The website can show these frames toa user of the mobile communications device 850. The website canregisters the user's taps, moves, or other interactions with the screenand send them to the CPU 806 via opened websocket connection stablishedbetween the mobile communications device 850 and the user terminal 890through a webserver 860.

The CPU 806 can receive the coordinates of the user interactions fromthe websocket connection and send them to the USB HID Module 808 via anI2C connection. The USB HID module 808 can receives the coordinates ofthe user interactions and send them to the primary computer 816 viastandard HID protocol.

While FIGS. 1-8 generally depict existing user terminals that can beretrofitted with contactless interaction capabilities, this disclosureis not so limited. Referring now to FIGS. 9-11, example embodiments aredepicted in which user terminals are manufactured with contactlessinteraction capabilities similar to the gesture-based and mobilecommand-based approaches described above.

Referring first to FIG. 9, a user terminal 990 that includes a built-ingesture processing unit 900 in accordance with one non-limitingembodiment is depicted. Similar to the user terminal 190 of FIG. 1, auser can provide input to the user terminal 990 via non-contactinggestures.

At the time of manufacture, the gesture processing unit 900 can beconnected to the peripheral modules of user terminal 990. The gestureprocessing unit 900 can have a gesture recognizer module 906 with a USBoutput that is to be treated as a peripheral device by the othercomputers on-board the user terminal 990. The user terminal 990 can alsobe built with one (or more) depth sensor camera 902 that is connected tothe gesture recognizer module 906. The gesture recognizer module 906 canbe connected to the primary computer of the user terminal 990 via theUSB port such that touch-based devices (touchscreen, keyboard, mouse,etc.) of the user terminal 990 can be replaced with a touchlessapproach.

An application running on the gesture processing unit 900 can activelymonitor for motion via the connected depth camera 902. Upon identifyingan “Initiate” gesture, the user terminal 990 can be configured tographically include a cursor on a device screen 970 to aid the user ingesture-based navigation. As describe above, gestures by the user can betranslated by the gesture recognizer module 906 to the equivalent HID(human interface device) commands, delivered to the primary computer viaUSB connection, and displayed on the screen 970.

Furthermore, while FIG. 9 schematically depicts a camera module 902having a depth sensor and a gesture recognizer module 906 having a TPU,this disclosure is not so limited. Similar to embodiments providedabove, for example, the camera module 902 does not necessarily need tohave a depth sensor, nor is a TPU required. As such, in some exampleembodiments, the camera module 902 can include a depth sensor and a TPUis not used for gesture processing. In other example embodiments, thecamera module 902 does not have a depth sensor and a TPU is used gestureprocessing.

Referring now to FIG. 10, an example user terminal 1090 is depicted thatincludes a built-in HID/Bluetooth module 1080 allowing for user inputfrom a coupled mobile communications device 1050. Similar to theembodiments described above, the user terminal 1090 can be configured toeither receive commands from the mobile communications device 1050 whenit is used as a trackpad, or the display of the user terminal 1090 canbe mirrored to the mobile communications device 1050 via a local networkconnection. In particular, the HID/Bluetooth module 1080 can be amicrocontroller with Bluetooth or WiFi module that connects as a HIDdevice to the main computer of the user terminal 1090. A user canconnect their mobile communication device 1050 to the HID/Bluetoothmodule 1080 through a Bluetooth-based connection to enable contactlessinteraction with the user terminal 1090.

Referring now to FIG. 11, an example user terminal 1190 is depicted thatincludes a built-in Internet-based mobile commands processing unitallowing for a bi-directional connection between a mobile communicationsdevice 1150 and the user terminal 1190 via a websocket protocol. Asnoted above, however, any suitable web-based communication protocol(s)can be utilized. The user terminal 1190 can be configured to allow auser to see a broadcast version of the user terminal 1190 displaymirrored on the mobile communications device 1150. Alternatively, theuser terminal 1190 can be configured to allow a user to use their mobilecommunications device 1150 as a touchpad for a display screen 11110 ofthe user terminal 1190.

When a user approaches the user terminal 1190 in its idle state, anapplication running on the user terminal 1190 can display an overlaymessage directing the user to navigate to a specific website address ontheir mobile communications device 1150. The display screen 1170 canalso show a unique QR code or other unique identifier. The user can thennavigate to the website address on their mobile communications device1150 and be prompted to scan the currently displayed QR code from thedisplay screen 1170, or otherwise enter the unique identifier. Forexample, in some embodiments, a unique alphanumeric code can bedisplayed on the display screen 1170 of the user terminal 1190 that canbe typed into the webpage displayed on the mobile communications device1150. Once the unique code is entered, a dedicated websocket connectioncan be established between the mobile communications device 1150 and theuser terminal 1190 through a webserver 1160. The display screen 1170 ofthe user terminal 1190 can be set to display alternative content,display a black screen, or display a message that private connection insession, for example.

In accordance with various embodiments, the user terminal 1190 canutilize a video compression algorithm and only send static images of thescreen to the mobile communications device 1150 to minimize the latencyand reduce the required bandwidth. The website presented on the mobilecommunications device 1150 can capture the user's interactions and sendthem to the web server 1160, which in turn sends them to the userterminal 1190. The user terminal 1190 can translate such interactions toan HID command.

It is noted that user terminals in accordance with the presentdisclosure can include multiple systems for different types ofcontactless interaction. By way of example, various user terminals maybe retrofitted or manufactured with units that allow users to provideinput via gestures (similar to FIGS. 1-3, and 9, for example), or theirmobile communications devices (similar to FIGS. 4-8 and 10-11, forexample). As such, a user wishing to interact with such user terminalcan optionally choose the type of non-contacting user interface theyprefer to use.

The processes described herein can be executed by programmableequipment, such as computers or computer systems and/or processors.Software that can cause programmable equipment to execute processes canbe stored in any storage device, such as, for example, a computer system(nonvolatile) memory, an optical disk, magnetic tape, or magnetic disk.Furthermore, at least some of the processes can be programmed when thecomputer system is manufactured or stored on various types ofcomputer-readable media.

It can also be appreciated that certain portions of the processesdescribed herein can be performed using instructions stored on acomputer-readable medium or media that directs a computer system toperform the process steps. A computer-readable medium can include, forexample, memory devices such as optical disk drives or hard disk drives.A computer-readable medium can also include memory storage that isphysical, virtual, permanent, temporary, semi-permanent, and/orsemi-temporary.

A “computer,” “computer system,” “host,” “server,” or “processor” canbe, for example and without limitation, a processor, microcomputer,minicomputer, server, mainframe, laptop, personal data assistant (PDA),wireless e-mail device, cellular phone, pager, processor, fax machine,scanner, or any other programmable device configured to transmit and/orreceive data over a network. Computer systems and computer-based devicesdisclosed herein can include memory for storing certain software modulesused in obtaining, processing, and communicating information. It can beappreciated that such memory can be internal or external with respect tooperation of the disclosed embodiments.

In various embodiments disclosed herein, a single component can bereplaced by multiple components and multiple components can be replacedby a single component to perform a given function or functions. Exceptwhere such substitution would not be operative, such substitution iswithin the intended scope of the embodiments. The computer systems cancomprise one or more processors in communication with memory (e.g., RAMor ROM) via one or more data buses. The data buses can carry electricalsignals between the processor(s) and the memory. The processor and thememory can comprise electrical circuits that conduct electrical current.Charge states of various components of the circuits, such as solid statetransistors of the processor(s) and/or memory circuit(s), can changeduring operation of the circuits.

The foregoing description of embodiments and examples has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or limiting to the forms described. Numerous modificationsare possible in light of the above teachings. Some of thosemodifications have been discussed, and others will be understood bythose skilled in the art. The embodiments were chosen and described inorder to best illustrate principles of various embodiments as are suitedto particular uses contemplated. The scope is, of course, not limited tothe examples set forth herein, but can be employed in any number ofapplications and equivalent devices by those of ordinary skill in theart.

1. A mobile commands processing unit enabling contactless interactionwith a user terminal, comprising: a video input port; a video outputport; at least one Universal Serial Bus (USB) port; a central processingunit (CPU); a short-range wireless communication module configured toreceive communications from a mobile communications device having atouchscreen interface, wherein the communications are based on userinteractions with the touchscreen interface; a human interface device(HID) module in communication with the CPU and configured to provide HIDcommands to a primary computer of [[a]] the user terminal via the USBport based on the communications received from the mobile communicationsdevice through the short-range wireless communication module; and avideo overlay module in communication with the CPU and configured toreceive source video from the primary computer of the user terminal viathe video input port and provide a modified video signal to a primarydisplay of the user terminal via the video output port based on thecommunications received from the mobile communications device throughthe short-range wireless communication module, wherein the modifiedvideo signal comprises an overlay of a cursor on the source video,wherein movement of the cursor on the primary display is responsive tothe user interactions with the touchscreen interface.
 2. The mobilecommands processing unit of claim 1, wherein the cursor is onlydisplayed during a contactless interaction session and is not displayedon the touchscreen interface of the mobile communications device duringthe contactless interaction session.
 3. (canceled)
 4. The mobilecommands processing unit of claim 1, wherein the communications arebased on user interactions with a touchscreen interface of a mobilecommunications device, and wherein the touchscreen interface functionsas a trackpad.
 5. (canceled)
 6. The mobile commands processing unit ofclaim 1, wherein the mobile commands processing unit is installable intoan ATM.
 7. The mobile commands processing unit of claim 1, wherein themobile commands processing unit is installable into a kiosk.
 8. Themobile commands processing unit of claim 1, wherein the mobile commandsprocessing unit is installable into a point of sale terminal.
 9. Amobile commands processing unit enabling contactless interaction with auser terminal, comprising: a video input port; a video output port; atleast one Universal Serial Bus (USB) port; a short-range wirelesscommunication module configured to receive communications from a mobilecommunications device having a touchscreen interface and a camera,wherein the communications are based on user interactions with thetouchscreen interface, wherein a video feed from the camera is providedto the touchscreen interface; a central processing unit (CPU) configuredto receive communications from the mobile communications device throughthe short-range wireless communication module and translate interactionswith the touchscreen interface to coordinates on a primary display of[[a]] the user terminal; a human interface device (HID) module incommunication with the CPU and configured to provide HID commands to aprimary computer of the user terminal via the USB port based on thecoordinates on the primary display of the user terminal; and a videooverlay module in communication with the CPU and configured to receivesource video from the primary computer of the user terminal via thevideo input port and provide a modified video signal to a primarydisplay of the user terminal via the video output port based on thecommunications received from the mobile communications device throughthe short-range wireless communication module, wherein the modifiedvideo signal comprises an overlay of a cursor on the source video,wherein movement of the cursor on the primary display is responsive tothe user interactions with the touchscreen interface.
 10. (canceled) 11.(canceled)
 12. (canceled)
 13. The mobile commands processing unit ofclaim 9, wherein the mobile commands processing unit is installable intoan ATM.
 14. The mobile commands processing unit of claim 9, wherein themobile commands processing unit is installable into a kiosk.
 15. Themobile commands processing unit of claim 9, wherein the mobile commandsprocessing unit is installable into a point of sale terminal.
 16. A userterminal enabling contact-based user interaction and contactless userinteraction via interactions with a mobile communications device havinga camera and a touchscreen interface, comprising: a power supply; aprimary computer; a primary display; and a mobile commands processingunit in communication with each of the primary computer and the primarydisplay, the mobile commands processing unit comprising a video outputport; at least one Universal Serial Bus (USB) port; a central processingunit (CPU) unit configured to receive communications from a webserver,wherein the CPU is configured to operate the user terminal in one of acontact-based operational mode and a contactless operational mode, andwherein a video signal provided to the primary display via the videooutput port is based on the communications received from a mobilecommunications device when operating in the contactless operationalmode, wherein communications from the mobile communications devicecomprise coordinates of a contact with a touchscreen interface of themobile communications device when the touchscreen interface isdisplaying an image of the primary display, and wherein the CPUtranslates the coordinates to the primary display; and a human interfacedevice (HID) module in communication with the CPU and configured toprovide an HID command to a primary computer via the USB port based onthe coordinates.
 17. The user terminal of claim 16, wherein the userterminal is an ATM.
 18. The user terminal of claim 16, wherein the userterminal is a kiosk.
 19. The user terminal of claim 16, wherein the userterminal is a point of sale terminal.
 20. A method of contactlessinteraction with a user terminal comprising; displaying, by a primarycomputer of the user terminal, a video signal on a primary display ofthe user terminal; presenting, on a touchscreen interface of a mobilecommunications device, a real-time image of the primary display capturedby a camera of the mobile communications device; receiving, by a mobilecommands processing unit of the user terminal, communications based onuser interaction with a mobile communication device, wherein thecommunications comprise coordinates of the interaction with thetouchscreen interface of the mobile communications device; translating,by the mobile commands processing unit of the user terminal, thecoordinates of the interaction with the touchscreen interface of themobile communications device to coordinates on the primary display ofthe user terminal, wherein an icon is graphically presented at thecoordinates on the primary display of the user terminal; and based onthe coordinates on the primary display of the user terminal, executing,by a primary computer of the user terminal, a command associated withthe icon.
 21. (canceled)
 22. The method of contactless interaction withthe user terminal of claim 20, wherein the user terminal is any of anATM, kiosk, and point of sale terminal.